Adds multimodal support and MMMU pro #675
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
need working will mainly be here, first is to have the greedy untill function working
| # TODO: What is the best option to pass images to the requests? | ||
| # dirty hack for now | ||
| for reqs in requests.values(): | ||
| for req in reqs: | ||
| req.specific = formatted_doc.specific |
There was a problem hiding this comment.
What is the best option to pass images to the requests?
There was a problem hiding this comment.
for now it would be using specifics but we could also add an images field to the request that default to None
There was a problem hiding this comment.
Hey @NathanHB, thanks for reviewing the PR. As we discussed internally, the only thing left is the tests, and the rest can be merged in the current state.
I also plan to conduct more experiments to evaluate VLM models after my vacation, and maybe, we will already have some user feedback to improve the evaluation.
I also left some comments, feel free to resolve them (if needed) while adding tests 🤗 Thanks!
|
hey @qubvel Taking care of adding tests and removing unneded stuff, thank you so much for the help in adding this feature :) |
NathanHB
changed the title
``` uv run lighteval accelerate "model_name=HuggingFaceTB/SmolVLM-Instruct" "lighteval|mmmu_pro|0|0" --use-chat-template --vision-model ``` --------- Co-authored-by: qubvel <[email protected]>
``` uv run lighteval accelerate "model_name=HuggingFaceTB/SmolVLM-Instruct" "lighteval|mmmu_pro|0|0" --use-chat-template --vision-model ``` --------- Co-authored-by: qubvel <[email protected]>
Aims to add multimodal support for transformers model by creating the
VLMTransformersModel.Adds the MMMU task
modify the prompt manager to support multimodal input (images for now)
adds a
lighteval accelerate vlmcli entry for creating the right config and using theVLMTransformersModeltests failing is because of shortcut it's normal
To test / use: